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Abstract. We discuss an extension of our method Image Inpainting Based on Coherence Transport. For the 
latter method the pixels of the inpainting domain have to be serialized into an ordered list. Up till now, to 
induce the serialization we have used the distance to boundary map. But there are inpainting problems 
where the distance to boundary serialization causes unsatisfactory inpainting results. In the present work 
we demonstrate cases where we can resolve the difficulties by employing other distance functions which 
better suit the problem at hand. 

Key words. Image Processing, Image Inpainting, Distance Functions 

AMS subject classifications. 65D18, 51K99 

1. Introduction. Non-texture image inpainting, also termed image interpola- 
tion, is the task of determining the values of a digital image for a destroyed, or con- 
sciously masked, subregion of the image domain. 

The simple idea of the generic single pass method - which forms the basis for our 
method Image Inpainting Based on Coherence Transport published in [ ] - is to fill the in- 
painting domain by traversing its pixels in an onion peeling fashion from the bound- 
ary inwards and thereby setting new image values as weighted means of given or 
already calculated ones. 

Telea has been the first to use such an algorithm in [ ]: the pixels are serialized 
according to their euclidean distance to the boundary of the inpainting domain and 
the weight is such that image values are propagated mainly along the gradient of the 
distance map. By his choices of the weight and the pixel serialization, the method of 
Telea is not adapted to image. 

In [4], we addressed the adaption of the weight: our method uses an image de- 
pendent weight such that image values are propagated along the estimated tangents 
of color lines which have been interrupted by the inpainting domain. That way we 
could improve the quality of the inpainting results compared to Telea (see figure 
1.1). Beyond that, we have illustrated in [ ] that our method matches the high level 
of quality of the methods by [3], [10], [' ], and [ ] while being considerably faster. 

But serializing the pixels by their distance to the boundary, as Telea and we in 
[4] did, is not always a good idea as figure 1.2 illustrates. The image in the middle 
shows the result which we obtain by using our method with distance to boundary 
serialization. The diagonal is not restored and the contours of the distance to bound- 
ary map indicate that this is due to the bad location of the skeleton which consists 
of the ridges of the distance map. Before reaching the skeleton, however, the perfor- 
mance is good and the diagonal is continued tangentially. The right image of figure 
1.2 shows the result which we get if we use another distance function (for pixel seri- 
alization) that suits the problem at hand better (see §3.1, figure 3.1). 

The way of serializing the pixels is an important degree of freedom and in this 
paper we use this freedom to resolve some obstructions which the distance to bound- 
ary serialization entails. The pixel serializations that we discuss are all induced by a 
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Fig. 1.1. Scratch removal. Left: vandalized image (courtesy ofTelea [ , Fig 8.i]), the white scratches are the 
inpainting domain. Middle: result ofTelea's method. Right: result of our method 




Fig. 1.2. A broken diagonal ? Left: data image with inpainting domain in white. Middle: result of our method, 
pixel serialization by distance to boundary, levels of the distance to boundary map are overlaid. Right: result of our 
method, pixel serialization by another distance function. 



more general type of distance function, so the new parameter of the method is in fact 
the user's choice of the distance function. 

Outline of the Paper. In section 2 we summarize the existing results and give a 
complete description of the basic algorithm. In section 3 we turn to concrete distance 
functions which are used to serialize the pixels of the inpainting domain. We describe 
three approaches: distance by harmonic interpolation (§3.1), modified distance to 
boundary (§3.2), and distance to skeleton (§3.3). Section 4 closes with a discussion. 

2. Summary of Existing Results. Our starting point is the generic algorithm 
for gray tone images. We assume that all gray tone images, seen as functions, take 
values in the real interval [0, 255] and we assume that gray tones are mapped onto 
[0,255] such that the natural order on the interval reflects the order of the shades 
of gray by their brightness from black (= 0) to white (= 255). Moreover, we will 
distinguish between discrete (digital) notions by using the index h and continuous 
(analog) notions where we omit the index. Continuous notions are thought of as the 
high-resolution limit of the corresponding discrete ones. Finally, we identify pixels 
with their midpoints. 

Notation. 

a) Oq^ is the image domain, the matrix of pixels for the final, restored image : 
CL 0/h -> [0,1]. 

b) Oft C O^/z is the inpainting domain whose values of have to be determined. 

c) n 0/ / z \O/ z is the data domain whose values of are given as W/z|n 0/j \n^ = ^o,/z- 

d) dClfr C 0^ is the discrete boundary, i.e., the set of inpainting pixels that have at 
least one neighbor in the data domain. 
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Continuous quantities are defined correspondingly. Finally, we define discrete and 
continuous e-neighborhoods by 

%z0) : = {y G n 0/h :\y-x\<e}, B e (x) := {y € O : \y - x\ < e} . 



Generic Algorithm. The basic idea is to fill the inpainting domain from its bound- 
ary inwards by using weighted means of given or already calculated image values. 
We assume that the pixels which are to be inpainted have already been serialized, i.e. 
O/j = {%i, %2, . . • , Xn) is an ordered list. The following set 

Kh( x k) -= B efh (x k )\{x kf ...,x N }, k = 1, . . . , N, 

denotes the neighborhood of the pixel x k consisting only of known or already in- 
painted pixels. Then, the algorithm reads as follows: 

u h\n 0fh \n h = u o,h / 

E w(x k/ y)u h (y) (2.1) 
u h (x k ) = r — , k = 

Here, w[x,y) > are called the weights of the algorithm and we assume that 

w[x,y) > , x eCl h . 



Pixel Serialization. In the generic algorithm, any serialization of the pixels can 
be used which geometrically goes from the boundary inwards in an onion peeling 
fashion. Now, let T : O — >• [0, T max ] be a map with T\y = 0, Y C 3Q, and which 
strictly grows into the interior of Q. We define the onion peels to be the level lines 
of T. Using such a map T or rather a discretized version : —> [0, T max ], we 
serialize the pixels of by 

T h (xj) < T h (x k ) => j<k 

into an ordered list = {x\, x<i,. .., x^). 

Clearly, there are different possible choices of a suitable T, but in the article [4] 
this degree of freedom has been fixed by setting T = d, where d is the euclidean 
distance to the boundary 3Q, 

d(x) = dist(x,9Q) , x e O . 



As pointed out in the introduction T = d is not always a good choice and in the next 
section we will use generalized distances T . 
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Weights. Here again, different choices for the weights are possible. We consider 
weights of the form 

w{x,y) = k (x, (x - y) • (2.2) 

with a kernel k{x,tj): 

a) Normal Transport (Telea's method). Telea, in [ ], uses the kernel 

7/ , \(N(x),n)\ . , ~\ t / \ VT(x) 
k(x,Tj) = 1 x I, / with N(x) - v 



|VT(x) 



Tough Telea has taken T = d, the kernel carries over to more general T. 
b) Coherence Transport (our method of [4]). For a given guidance field called g we use 
the kernel 

k A x >V) = /I Fexp (-^ (^(x),^) 2 ^) . (2.3) 

The guidance vector g(x), supposed to be an approximate tangent, is computed 
by employing structure tensor analysis. The set-up of the structure tensor S a ,p{x) 
is as follows: 



0C(j(y,x) = J K a (y — h) dh 

O(x) 

My>x) = ^ • / K a (y - h) - u(h) dh 

O(x) 

S<r /P (x) = ^ ^ • J K p (x-y)- V y v a {y,x) • V y vl{y,x) dy 



O(x) 

where Q(x) := {y G O : T(y) < T(x)} U Oo\0 and K a is a Gaussian kernel. The 
coherence vector g(x) which enters k^ as guidance is the eigenvector of S (7/ p(x) 
w.r.t. the minimal eigenvalue. Since g depends on the image u, the weight does 
here, too. 

For more details on the usage of the structure tensor w.r.t. coherence transport 
inpainting see [ ] and [6]. For other applications of structure tensor analysis and 
the related scale space theory see e.g. [1], [14] and [2]. 

A Note on the Theory Behind. Because continuous theory is not an issue of this 
paper, we want at least make some comments. For weights of the form (2.2) we have 
shown in [4] and [6] that the high-resolution (h —> 0) vanishing-viscosity (e —> 0) 
limit yields transport/ advection PDEs of the form: 

a) Normal Transport. 

(N(x), Vu(x)) = in Q\E , u\ m = u \ m . 

b) Coherence Transport. 

(cp(x),Vu{x)) =0 inO\E, u\ m = u \ m , (c H (x),N(x)) > > . 

(2.4) 
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In both cases the set E denotes the set where N(x) = VT(x)/| VX(x)| is singular. 
The side condition (c^(x), > ft^ is the continuous counterpart of the pixel 

serialization: the function T here plays the role of a Lyapunov function and this 
condition says that the characteristics of the advection PDE evolve strictly into the 
interior of Q and stop at E. The general existence and well-posedness theory for such 
type of equations can be found in [8] and [7]. This theory is applied to the analog 
inpainting model above in [6, chapter 6]. 

Finally, in order to better understand the role of the parameter ]i of the kernel 
kji, we cite two results of [4]. Firstly, according to [ , theorem 1] every weight w of 
the form (2.2) corresponds to a transport field c. Secondly, in [ , theorem 2], when 
using the coherence transport kernel k^, we have proved an asymptotic expansion 
ofcpi(x), w.r.t. ]i oo, which implies the following limit behavior 

(g(x) ,{g(x),N(x))>0 
lim = I -g(x) ,(g{x),N{x)) < . (2.5) 
(N(x) ,(g(x),N(x))=0 

That means the coherence vector g(x) guides in fact the coherence transport (2.4) and 
]i is the strength of this guidance. 

Interface of the Algorithm. In the next section the algorithm will be performed 
only in its coherence transport version. That is, the weight function w has the form 
of (2.2) with the kernel k^ given by equation (2.3). The execution of the coherence 
transport algorithm depends on the choice of four parameters: 

• £, the averaging radius, 

• ]i, the guidance strength of k ji/ 

• a and p, the scale parameters of the smoothing operations in the structure 
tensor S a/P . 

Finally, the algorithm is supplied with the data image uq and a sorted list of the pixels 
which are to be inpainted. Any item of this list has the form 

[i 1 T h(hj)\, 

whereas are the pixel coordinates. The list is sorted in ascending order of the 
Thd'j) values. 

3. Concrete Distance Functions. The generic algorithm of section 2 depends on 
a prescribed pixel serialization, which orders the pixels from the boundary inwards. 
In all previous experiments of [ ] the pixels were serialized by their euclidean dis- 
tance to boundary. The advantage in practice is that for all types of domains the 
approximate distance to boundary map is easy to generate by the Fast Marching 
Method (see [ ] and [ ]). But the disadvantage is that it is not always the best choice 
if one wants to get a nice looking inpainting result. Here, we present three other ways 
of setting up a generalized discrete distance function T^. We show a few synthetic 
examples where they yield better inpaintings than the distance to boundary. 

3.1. Distance by Harmonic Interpolation. The broken diagonal is our first ex- 
ample. Figure 3.1 (b) shows the result when using the distance to boundary d^. The 
desired inpainting result would be the restored diagonal but the diagonal is only 
partly continued correctly. Figure 3.1 (c) - with the contours of d^ overlaid - demon- 
strates that the undesired effect is due to the wrong location of the stop set. Figure 3.1 



6 



T. MARZ 




(a) damaged image (b) inpainted image (c) with contours of d h 




(d) damaged image (e) inpainted image (f) with contours of T h 



Fig. 3.1. Broken diagonal. In the first row we have (a) the damaged image with the inpainting domain n& in 
white, (b) the result inpainted using euclidean distance to boundary dh, and (c) the result with contours of dh. In 
the second row we have (d) the same damaged image with a specified stop set in red, (e) the result inpainted using 
distance by harmonic interpolation Th, and (f) the result with contours ofT^. 



(e) shows the result when using which we obtain by prescribing a better located 
stop set and harmonic interpolation. Here, the algorithm is able to restore the diago- 
nal. For both cases we have used the same set of parameters [e, ]i, a, p] = [3, 50, 0.5, 5] . 

Now, we describe the construction of the discrete distance function T^. Here, we 
can prescribe the location of the stop set arcs and their distance values. 

Because the boundary is the start set, the discrete distance function 7^ shall 
equal zero on 30^. In addition, we take at least one or more discrete curves Y\, 
k G {!,... ,n} , which are contained in and should belong to the stop set E^. 
Moreover, for every we specify a distance value > 0. The remainder of 
then, is calculated by harmonic interpolation. That is, we solve the discrete Laplace 
equation 

n 

A h T h = in Ci h \L h , Z h = U T k h , 

k=l 

Th = on , T h = t k on l\ , k = 1, . . . , n . 

Hereby, the discretization of the Laplacian is due to the five-point-stencil 

A h T h (i,j) = T h (i - l,j) + T h (i,j - 1) - 4tT h (i,j) + T h (i,j + 1) + T h (i + . 

Since harmonic interpolation provides a minimum and maximum principle, our 
construction of can be imagined as the setting up a tent roof over the domain 
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where are the locations of the tent poles, and every tent pole of Y^ has the length 
t k . 

Unfortunately, not every choice of curves Y\, with time values t^, results in a 
valid distance function: T^ might have local minima and so would not strictly in- 
crease into the interior of fi^. In the case of a single curve Y^ the resulting is a 
valid distance because of the minimum and maximum principle, see figure 3.2. If 
there are two or more curves Y\, the question whether the resulting is valid or not 
depends on the location of the curves in relation to each other and the differences of 
their prescribed values t\ . Figure 3.3 (b) shows an example with three curves l\, Y\ 
and Y\ with 

h = t 2 = 250 > t 3 = 50 . 

Here, all points of Y^ are local minima of T h . But, if we keep the geometry of l\, Y\ 
and Y\, and change the prescribed distances to 

h = t 2 = 250 > t 3 = 249 , 

then does not have any minima (see figure 3.3 (c)). So, the resulting is admissi- 
ble. Generally speaking, if we have two or more curves Y\, with different prescribed 
distances t^, and if the values are chosen unfavorably, then the resulting T^ might 
possess local minima on some of the T^. 

Let us review our first example. Figure 3.1 (e) illustrates that our T^ - which 
is that shown in figure 3.2 (b) - works fine. More generally, if we think of 1^ as 
an initial scratch, which has been dilated to all of fi^ over the time T^, then the 
backward filling-in process, if 1^ is well located, makes the matching opposite sides 
come together. Clearly, if we deliberately place 1^ badly, then the method must fail 
(see figure 3.4, the parameters are [e, ]i, a, p] = [3, 50, 0.5, 5], as before). 

Now, we discuss two further examples. Figure 3.5 (a) shows two broken diag- 
onals with the same inpainting domain as in figure 3.1 (a). We emphasize here that 
the appearance of an undesired effect depends on how the edge that needs to be 
continued is located in relation to the inpainting domain. In figure 3.5 (b) - which 
is inpainted using distance to boundary - the bottom-left-to-top-right diagonal is 
continued as desired, while the continuation of the top-left-to-bottom-right diagonal 
suffers from a badly located stop set. In Figure 3.5 (d) we have the damaged image 
with the single curve Y^, t\ = 127 shown in red. Figure 3.5 (e) shows the result in- 
painted using the distance by harmonic interpolation. Again, a good location of 



8 



T. MARZ 





(c ) valid 1 h 



Fig. 3.3. (a) shows the domain Cl h and the desired stop arcs, red: l\ and Y\, blue: Y\. For (b) the distances 
on have been set to t\= t^= 250, t$ — 50 and we can see that the resulting has as local minimum and is 



thus not valid. For (c) the distances on have been set to t\ 



■ t2 = 250, to, = 249 and resulting is valid. 





(a) damaged image (b) inpainted image (c) with contours of T h 

Fig. 3.4. Broken diagonal; method must fail if the stop set is deliberately placed badly, (a) the damaged image 
with the inpainting domain f\ in white with a single stop arc in red, (b) the result inpainted using distance by 
harmonic interpolation and (e) the result with contours ofT^. 



2^ makes for a good result. For both cases we have used the same set of parameters 
[e,ji,(r,p\ = [3,50,0.5,5]. 

Figure 3.6 (a) shows a damaged cross junction. A cross junction would, in any 
case, geometrically be the simplest object for completion. Using the distance to 
boundary we obtain the result shown in figure 3.6 (b). Here, the stop set which is 
the central arc of the skeleton (see figure 3.6 (c)), has an unfavorable location because 
the bar coming from the right-hand side can never reach its counterpart. Setting as 
the center of the cross junction (see figure 3.6 (d)) and using the distance by harmonic 
interpolation, we are able to restore the cross junction (see figure 3.6 (e)). Which of 
the bars is closed in the end depends on the coherence strength. The brighter bar 
has the higher contrast w.r.t. the black background and is thus of stronger coherence. 
This is the reason why this bar is closed. For both cases we have used the same set 
of parameters [e,}i,a,p\ = [5,100,0.5,10]. 

3.2. Modified Distance to Boundary. Figure 3.7 (a) shows a damaged stripe pat- 
tern. Performing the algorithm with distance to boundary and the set of parameters 
[t,}i,a,p\ = [5,100,0.5,10] yields the result shown in figure 3.7 (b). The difficulty, 
is that the coherence vector which is tangent to the edges is orthogonal to the lower 
left and the upper right segment of dCl^. Thus, the transport vector c switches to the 
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(a) damaged image (b) inpainted image (c) with contours of d h 




(d) damaged image (e) inpainted image (f) with contours of T h 



Fig. 3.5. Two broken diagonals. In the first row we have (a) the damaged image with the inpainting domain 
O/j in white, (b) the result inpainted using euclidean distance to boundary d^, and (c) the result with contours of d^. 
The result in (b) demonstrates that the undesired effect depends on how edges are located in relation to the inpainting 
domain. In the second row we have (d) the same damaged image with a specified stop set in red, (e) the result 
inpainted using distance by harmonic interpolation T h , and (f) the result with contours ofT h . 



normal N as in the exceptional case of equation 2.5. Figure 3.7 (c) confirms the switch 
of the transport to N. 

To combat this we suggest a modification of the distance map set-up. The eu- 
clidean distance to boundary map d is the viscosity solution of 

|W|=1 in Q, d\ m = 0. 

The modification is that we search for the euclidean distance to a subset T of the 
boundary 3Q, i.e., 

|W*| = 1 in O, d*\i = . 

We classify the points which shall not belong to T. Assume x G 30 satisfies 
(x) = 0, then the boundary normal is given by N(x) = W* (x). Now, if we have 
at x 

{ g (x),N(x)) 2 = 0, 

whereas g is the guidance vector, then either there is no guidance (\g(x) | = 0) or the 
guidance vector does not point inwards. Such a boundary point shall not belong to 
T. In fact, we use the stronger condition 



0<(g(x),N(x)) 2 <<y 
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Fig. 3.6. Broken cross junction. In the first column we have (a) the damaged image with the inpainting domain 
Clh in white, (b) the result inpainted using euclidean distance to boundary d^, and (c) the result with contours of 
dfr. In the second column we have (d) the same damaged image with the stop set being a single point in red (at the 
center of the cross junction), (e) the result inpainted using distance by harmonic interpolation T h , and (f) the result 
with contours ofT^. 



with a threshold parameter < 7 < 1. Complementarity, the set of active boundary 
points r is given by 

r = {xG3Q:(g(x),N(x)) 2 >7}. 

Clearly, the new parameter 7 must be chosen such that T is not empty 

We have applied this idea to the stripe pattern: the red lines in figure 3.7 (d) are 
the active boundary points and the result is shown in figure 3.7 (e). Our standard 
parameters are [e,ji,a,p] = [5,100,0.5,10], while the additional parameter is set to 
7 = 0.1. The discrete approximation d^^ was computed using the fast marching 
method. The overlaid contour plot of d* /H in figure 3.7 (f) shows that the guidance 
vector always points inwards. 

3.3. Distance to Skeleton. The third approach to obtaining a serialization is to 
use the distance to a prescribed stop part of the skeleton. Let S k , k G {1, . . . , n} be 
curves in the image domain fig. Those curves S k which are contained in O will later 
belong to the skeleton S. Let, then, X* be the viscosity solution of 



VT*| =1 in Qq , 



T*=0 on S k , fcG {!,..., n}, 
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(a) damaged image (b) inpainted image (c) with contours of d h 




(d) red active boundary T (e) inpainted image (f) with contours of d^h 



Fig. 3.7. Stripe pattern. The difficulty is that the damage is aligned with the pattern. In the first row we have 
(a) the damaged image with the inpainting domain Ci h in white, (b) the result inpainted using euclidean distance to 
boundary d^, and (c) the result with contours of d^. In the second row we have (d) the same damaged image with 
the active part Y of the boundary in red, (e) the result inpainted using d*^ which is the euclidean distance to T, and 
if) the result with contours of d^^. 

and let 

T*,max = (x) • 

The desired distance function is defined by 

T(x) = T* /max - T*0) , x e O . 

Warning: as in the case of harmonic interpolation (see §3.1) one must check if T is 
admissible, i.e., if T is free of local minima. 

Figure 3.8 shows the result for the example of the broken diagonal. The red 
curve in figure 3.8 (a) defines (discrete) which shall belong to the skeleton. The 
discrete approximation T*^ was computed using the fast marching method and the 
set of parameters is [e,ji,a,p] = [3,50,0.5,5], the same as we used for figure 3.1 
(e). The inpainted result here (figure 3.8 (b)) is the same as in figure 3.1 (e), but 
the distance function has changed (compare figures 3.8 (c) and 3.1 (f)). If there is 
only one curve which is completely contained in O, then distance by harmonic 
interpolation (with l\ = Si) and distance to skeleton will yield very similar results. 
If there are two or more curves, then distance by harmonic interpolation allows for 
different distance values on Y\, while distance to skeleton has exactly one distance 
value on all of the curves S\. In contrast to distance to skeleton, the distance by 
harmonic interpolation method requires Th\dQ h = 0- 
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(a) damaged image (b) inpainted image (c) with contours of T h 

Fig. 3.8. Broken diagonal, (a) shows the damaged image, white O^, with the prescribed skeleton arc S\ in 
red. (b) is the result inpainted using distance to skeleton T^. (c) is the result with contours ofT^. 




(a) damaged image (b) inpainted image (c) with contours of T h 



Fig. 3.9. Stripe pattern, (a) shows the damaged image, white f\, with the red arc outside f\. (b) is the 
result inpainted using T^, the distance to "skeleton" S\. (c) is the result with contours ofT^. 

Moreover, since X* is defined on Qq, we can place S k outside of the inpainting 
domain Q. We use this possibility to restore the stripe pattern in figure 3.9. The 
parameters are [e, ]i,a,p\ = [5,100,0.5,10], the same as we used for figure 3.7. It is 
obvious from the level lines of (see 3.9 (c)) that the guidance vector always points 
inwards. Thus, we get the desired result again here. 

Remark: The construction here, as well as that of §3.2 produces distance func- 
tions whereas not the whole boundary belongs to the start set. Besides that, it is 
possible that parts of the boundary belong in fact to the stop set. 

4. Discussion. In the previous section we have demonstrated that there are in- 
painting problems where a distance to boundary induced serialization of the pixels 
is not favorable and we have constructed other distances which are able to resolve 
the difficulties which appear. 

So far we have considered synthetic images, because their image geometry is 
easy to understand. Thus, we were able to construct distances which are adapted to 
the image or rather to an expected result. When we face natural inpainting problems, 
as shown in figures 1.1 (left) and 4.1 (a), it is not as easy to set up an adapted distance 
function. This is because 

• the geometry of the image is harder to understand, 

• the damaged region is complicated. 
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(a) original image (courtesy of [3, figure 6]) (b) inpainted: [e, ]i, a, p] = [4, 25, 2, 3] 



Fig. 4.1. Removal of superimposed text. The letters in (a) are the inpainting domain f\ 

Moreover, if the damaged region Q consists of many connected components and one 
wants to use our distance by harmonic interpolation technique (§3.1 ), one must pre- 
scribe arcs l\ for every single component of connectivity; for the example of figure 
4.1 (a) that means for every single letter. This can be time consuming (for the user, 
not for the computer). 

In contrast, the distance to boundary map can be computed fast and easily for 
every type of inpainting domain. And, inpainting using distance to boundary often 
produces results of high quality when applied to natural inpainting problems (see 
figures 1.1 (right) and 4.1 (b)). This is because for natural inpainting problems the 
damage often is such that color lines have been broken by scratches (By scratches we 
mean rather thin and lengthy damages). If the damage is of this type, the skeleton 
of Q, being a simplified version of the scratch, is well placed and so the filling-in 
process makes the matching opposite sides come together. The images shown in 
figures 1.1 (right) and 4.1 (a), like many other natural inpainting problems, have this 
type of damage. Thus, our inpainting method using distance to boundary is able to 
produce results pleasing to the eye. 

5. Acknowledgements. The author would like to thank Folkmar Bornemann 
for his advice and the inspiring discussions. 
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